Repository navigation
Run the tooling tests and the suites under a second thread environment - #11
Merged
Merged
Conversation
Two of them ran in no workflow, and both had the shape the verification gate had before 845a4f2: a check nobody runs does not report a regression late, it reports nothing. `verification/tests` is the suite guarding the gate itself -- the comparison logic, the record selector, the inventory readers, the GIL-region and thread-count scans, the move bijection. 114 tests, run by hand only. Two of them are hardcoded tripwires, the `ALL` set in test_record_selector and the Makefile target spelling in test_wrap_link_guard, and a tripwire nobody runs is a comment. The other is the thread environment. Every CI run of the C suite has used the runner's default of two threads, and the Python legs run with OMP_NUM_THREADS unset, which tests/conftest.py already records as the condition that lets every test claim every core and makes several thrash. Two real defects in the parallelism work appeared only under a non-default cap: an assertion whose two outcomes collided on the same number, and a policy clause reading omp_get_num_procs() where it needed omp_get_max_threads(), which planned sixteen threads under OMP_NUM_THREADS=2. One environment cannot find that class. So the C suite now also runs at one thread and at three. One is where a reduction or a per-thread buffer can be skipped entirely; three is odd, divides no chunk count evenly, and -- unlike another power of two -- cannot collide with the plan fallback that made the test above pass by luck. At one thread the suite reports 438 passed and 1 skipped: test_plan.c:607 needs two threads to tell "the worker inherited the plan" from "the worker fell back", and skipping is the right answer there. The new tests assert the property rather than the YAML spelling, and each was mutation-tested: commenting out the tooling step, moving --threads 1 to 2, moving --threads 3 to 4, and dropping the OMP_NUM_THREADS pin each fail exactly one test and leave the other six passing. They prove a command is present outside a comment, not that the job is reached or that it passed -- the same limit the thread-plan wiring line turned out to have, stated in the file rather than discovered later.
The gate grew a line in 7f3717e and the table here never gained the row, so the one check that would have caught a region running at one thread per core regardless of the plan was undocumented in the file that explains the gate. Two counts beside it had drifted with the suites they describe: the C half is 439 tests, not 331, and `exact` holds 416 quantities, not 351. Neither number is recorded anywhere the gate reads, which is why they could drift in the first place.
The new thread-environment job failed on its first run, which is the job working: under OMP_NUM_THREADS=3 two tests in test_runtime.py failed, one asserting the plan never exceeds availability and one asserting a pinned BEAM_AVAILABLE_CORES drives the plan. Neither is a defect in BEAM. Clause 2 of the policy (src/native/core/plan.c) treats OMP_NUM_THREADS as the caller deciding, exactly as set_threads() does, so the runtime's own answer stands and availability does not revise it -- a plan of 3 against an availability of 2 is the documented behaviour, and under a cap of 16 the plan is 16, wider than max_width, for the same reason. Both tests describe clause 3, the path that reads the free cores, and neither said so. They passed for years because no run had ever set the variable: a test whose precondition is "the environment nobody varied" is not wrong, it is silently narrow, and the fix is to state the precondition rather than to weaken the assertion. So each skips when the plan reports omp_num_threads, which is the value the policy actually read rather than a second guess at os.environ. This is how the C side already does it: test_the_default_scope_stands_aside_for_omp_ num_threads skips unless the variable is set at process start, and pins availability to 64 to prove clause 2 ignores it. Between them the two clauses are covered in the environments where each applies -- clause 3 on the six matrix legs where the variable is unset, clause 2 on the C side and in the new job. The pinned-availability test keeps its first assertion unconditional: that BEAM_AVAILABLE_CORES is observable in the plan holds under every clause, and only its effect on the width is clause 3's business. Verified both ways: under OMP_NUM_THREADS=3 the two skip with their reasons and the file's other 12 tests pass; with the variable unset both run and pass, so the guard is conditional rather than a disable. The test ids do not move -- collection is byte-identical to pytest_cases.txt at 1243.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two checks this repository owns ran in no workflow. Both had the shape the
verification gate had before #8: a check nobody runs does not report a
regression late, it reports nothing.
verification/testsnow runs in CI114 tests guard the gate itself — the comparison logic, the record selector,
the inventory readers, the GIL-region and thread-count scans, the move
bijection. They ran by hand only. Two of them are hardcoded tripwires (the
ALLset intest_record_selector, the Makefile target spelling intest_wrap_link_guard), and a tripwire nobody runs is a comment.They stay outside
tests/deliberately: CI copiestests/elsewhere and runsit against the installed package, so a test that imports this repository's
tooling cannot live under it. The new
Verification toolingjob builds the Cbinaries first, so the three
test_build_dependenciescases that decline tojudge a stale build actually run.
The suites now run under more than one thread environment
Every CI run of the C suite has used the runner's default of two threads, and
the six Python legs run with
OMP_NUM_THREADSunset — whichtests/conftest.pyalready records as the condition that lets every testclaim every core and makes several thrash rather than run.
Two real defects in the parallelism work appeared only under a non-default
cap: an assertion whose two outcomes collided on the same number, and a policy
clause reading
omp_get_num_procs()where it neededomp_get_max_threads(),which planned sixteen threads under
OMP_NUM_THREADS=2. One environmentcannot find that class.
Thread environmentsruns the C suite at 1 and at 3, and the Python suite at3. One thread is where a reduction or a per-thread buffer can be skipped
entirely. Three is odd, divides no chunk count evenly, and — unlike another
power of two — cannot collide with the plan fallback that once made that test
pass by luck.
At one thread the C suite reports 438 passed and 1 skipped:
test_plan.c:607needs two threads to tell "the worker inherited the plan" from "the worker fell
back", and skipping is correct there.
What the new tests prove, and what they do not
They assert the property, not the YAML spelling, and each was mutation-tested
against the real workflow: commenting out the tooling step, moving
--threads 1to 2, moving--threads 3to 4, and dropping theOMP_NUM_THREADSpin each fail exactly one test and leave the other sixpassing. A commented-out job does not count as present.
They prove a command is present outside a comment. They do not prove the
job is reached, that its
if:lets it run, or that it passed — the same limitthe thread-plan wiring line turned out to have. That limit is stated in the
test file rather than left to be discovered.
Verified locally
byte-identical, including the 78 numeric keys, the 58 bit-identical math
values and the 416 exact quantities
No C or Python source changed, so no baseline moved and the extension is
untouched.
Also
The gate grew a
thread countsline in 7f3717e and the table inverification/README.mdnever gained the row — the one check that catches aregion running at one thread per core regardless of the plan was undocumented
in the file that explains the gate. Added, along with two counts that had
drifted with the suites they describe (the C half is 439 tests, not 331;
exactholds 416 quantities, not 351). Neither number is recorded anywhere thegate reads, which is why they could drift.